Multimodal contrastive learning for unsupervised video representation learning

In this paper, we propose a multimodal unsupervised video learning algorithm designed to incorporate information from any number of modalities present in the data. We cooperatively train network corresponding each modality: at stage training, one these networks is selected be trained using output other networks. To verify our algorithm, model RGB, optical flow, and audio. then evaluate effectiveness by performing action classification nearest neighbor retrieval on supervised dataset. compare triple modality contrastive models or two modalities, find that all three tandem provides 1.5% improvement UCF101 accuracy, 1.4% R@1 recall, 3.5% R@5 2.4% R@10 recall as compared only RGB demonstrating merit utilizing many possible cooperative model.